Image processing device, image display device, and program

ABSTRACT

Provided is an image processing device, image display device, and program that allow an image captured by an imaging device not having a vibration suppression function or an image whose vibration has been suppressed incompletely to be displayed on a display device with the vibration suppressed. The image processing device includes a motion estimator configured to estimate the amount of motion of an object between a first image and a second image later than the first image and a motion compensator configured to perform a conversion process on the second image so that vibration of the object between the first image and the second image is suppressed, on the basis of the amount of motion of the object. The motion estimator has a first estimation mode in which the amount of motion of the object is estimated in a predetermined search area and a second estimation mode in which the amount of motion of the object is estimated in a larger area than the search area in the first estimation mode.

TECHNICAL FIELD

The present invention relates to an image processing device, imagedisplay device, and program that suppress the vibration of an image.

BACKGROUND ART

To suppress the vibration of an image, various image processing methodshave been proposed.

To provide an imaging device that is able to prevent a reduction in theperformance of a blur correction mechanism and to easily track a subjecteven if the subject moves out of the angle of view, Patent Literature 1discloses an imaging device that includes an image sensor configured tocapture subject light guided by an imaging optical system and to outputa captured image and an optical correction system (e.g., a correctionlens) for correcting an image blur caused by vibration applied to thedevice, wherein the imaging device tracks the subject in the capturedimage by selecting one of an optical tracking mode in which the subjectis tracked by driving the optical correction system and an electronictracking mode in which the subject is tracked by cutting out the area ofthe subject from the captured image, in accordance with whether apredetermined criterion is satisfied.

CITATION LIST Patent Literature

[Patent Literature 1] Japanese Unexamined Patent Application PublicationNo. 2016-46580

SUMMARY OF THE INVENTION Technical Problem

According to the imaging device of Patent Literature 1, the vibration(blur) of an image can be suppressed by adjusting the zoom position ofthe correction lens. In the case where an image captured by an imagingdevice not having a vibration suppression function is displayed on adisplay device, the vibration of the displayed image remainsunsuppressed. Also, even an imaging device having such vibrationsuppression function may not be able to completely suppress thevibration of an image. So, when an image in which vibration has not beencompletely suppressed is displayed on the display device, the vibrationof the displayed image remains.

The present invention has been made in view of the foregoing, and anobject thereof is to provide an image processing device, image displaydevice, and program that allow an image captured by an imaging devicenot having a vibration suppression function or an image whose vibrationhas been suppressed incompletely to be displayed on a display devicewith the vibration suppressed.

Solution to Problem

The present invention provides an image processing device including amotion estimator configured to estimate the amount of motion of anobject between a first image and a second image later than the firstimage and a motion compensator configured to perform a conversionprocess on the second image so that vibration of the object between thefirst image and the second image is suppressed, on the basis of theamount of motion of the object. The motion estimator has a firstestimation mode in which the amount of motion of the object is estimatedin a predetermined search area and a second estimation mode in which theamount of motion of the object is estimated in a larger area than thesearch area in the first estimation mode.

According to the present invention, the motion estimator configured toestimate the amount of motion of the object between the first image andthe second image later than the first image has the first estimationmode in which the amount of motion of the object is estimated in thepredetermined search area and the second estimation mode in which theamount of motion of the object is estimated on the basis of the largerarea than the search area in the first estimation mode. For this reason,images are displayed in the first estimation mode, in which calculationis small in amount, and when it is difficult to estimate the amount ofmotion of the object in the first estimation mode, the first estimationmode is switched to the second estimation mode. Thus, the vibration ofthe image can be efficiently suppressed on the display device sidewithout significantly consuming the calculation resources. As a result,an image captured by an imaging device not having a vibrationsuppression function or an image whose vibration has been suppressedincompletely can be displayed on a display device with the vibrationsuppressed.

Various embodiments of the present invention are described below. Theembodiments below can be combined with each other.

Preferably, when a predetermined criterion is satisfied, the motioncompensator performs the conversion process on the basis of the amountof motion estimated in the second estimation mode, and when thepredetermined criterion is not satisfied, the motion compensatorperforms the conversion process on the basis of the amount of motionestimated in the first estimation mode.

Preferably, in the first estimation mode, the motion estimator defines,as an estimation error, a value calculated on the basis of a differencein luminance between the object included in the first image and apredetermined area included in the second image, and the predeterminedcriterion is that the estimation error exceeds a predetermined value, orthat the estimation error exceeds a predetermined multiple of an averagevalue, or both.

Preferably, the second estimation mode is a global motion estimationmode in which the amount of motion of the object is estimated from theentire first and second images.

Preferably, in the second estimation mode, the amount of motion of theobject is estimated in a new search area obtained by expanding a size ofthe search area in the first estimation mode.

Preferably, the motion estimator estimates the amount of motion of theobject using pattern matching or a feature vector.

Preferably, the image processing device further includes an area setterconfigured to set an area corresponding to the object in the firstestimation mode.

Preferably, the motion compensator performs the conversion process onthe second image so that the area set by the area setter is located inan approximate center of a display surface on which the first and secondimages are displayed.

Preferably, the image processing device further includes a mode switchconfigured to switch between the first estimation mode and the secondestimation mode.

Preferably, when the predetermined criterion is not satisfied, the modeswitch switches the second estimation mode to the first estimation mode.

Preferably, an image display device including any one of the imageprocessing devices is provided.

Preferably, there is provided an image processing program for causing acomputer to function as a motion estimator configured to estimate theamount of motion of an object between a first image and a second imagelater than the first image and a motion compensator configured toperform a conversion process on the second image so that vibration ofthe object between the first image and the second image is suppressed,on the basis of the amount of motion of the object. The motion estimatorhas a first estimation mode in which the amount of motion of the objectis estimated in a predetermined search area and a second estimation modein which the amount of motion of the object is estimated in a largerarea than the search area in the first estimation mode.

BRIEF DESCRIPTION OF DRAWINGS

FIGS. 1A to 1C are conceptual drawings showing the vibration of animage, in which images are inputted in the order of 1A, 1B, and 1C; andan arrow represents time t (the same applies to the following drawings).

FIG. 2 is a diagram chronologically showing the functional configurationof an image processing device 1 according to a first embodiment of thepresent invention and images inputted to the image processing device 1.

FIG. 3 is a flowchart showing an example of a process performed by theimage processing apparatus 1 according to the first embodiment of thepresent invention.

FIGS. 4A to 4C are conceptual diagrams showing an aspect in which alocal area tracking unit 4 tracks a local area LA, in which FIG. 4Ashows an input image 1; FIG. 4B shows an aspect of tracking of the localarea LA in a search area SA; and FIG. 4C shows an input image 2.

FIGS. 5A and 5B are graphs showing a predetermined criterion.

FIG. 6 is a diagram showing the basic concept of a local motionestimation mode (an example of a first estimation mode) according to oneembodiment of the present invention.

FIGS. 7A to 7F are conceptual diagrams showing an aspect in which amotion compensator 8 compensates for the position of an image in thelocal motion estimation mode so that the local area LA is displayed inthe approximate center of the display surface, in which a hatchedportion represents an area where no image is displayed.

FIG. 8 is a diagram showing the basic concept of a global motionestimation mode (an example of a second estimation mode) according toone embodiment of the present invention.

FIG. 9 is a conceptual diagram showing the global motion estimationmode, in which arrows represent the movement directions of divided areasobtained by dividing an image; and the movement directions of sixdivided areas of the nine divided areas are an upper-right direction andtherefore the entire element is estimated to be moving in theupper-right direction.

FIGS. 10A to 10H are conceptual diagrams showing an aspect in which themotion compensator 8 compensates for the position of an image in theglobal motion estimation mode so that an object is displayed in theapproximate center of the display surface, in which a hatched portionrepresents an area where no image is displayed.

FIG. 11 is a flowchart showing an example of a process performed by animage processing device 1 according to a second embodiment of thepresent invention.

FIGS. 12A to 12C are conceptual diagrams showing an aspect in which theimage processing device 1 according to the second embodiment of thepresent invention expands the size of a search area SA in the localmotion estimation mode and thus a first estimation mode is switched to asecond estimation mode (this process corresponds to S7 in FIG. 11).

FIG. 13 is a flowchart showing an example of a process performed by animage processing device 1 according to a third embodiment of the presentinvention.

DESCRIPTION OF EMBODIMENTS

Now, embodiments of the present invention will be described withreference to the drawings. Various features described in the embodimentsbelow can be combined with each other.

First Embodiment

Referring now to FIGS. 1 to 10, an image processing device 1 accordingto a first embodiment of the present invention will be described.

Vibration of Object in Image

As shown in FIGS. 1A to 1C, captured images may vibrate due to camerashake or the like, resulting in vertical or horizontal displacement ofan object in the images. As used herein, the term “images” refers tomultiple images arranged in chronological order and, in particular,moving images.

For example, an image shown in FIG. 1A is displaced downward in thesubsequent frame shown in FIG. 1B. Also, the image shown in FIG. 1B isdisplaced in an upper-right direction in the subsequent frame shown inFIG. 1C. As seen above, an object (a broken-line area in FIG. 1)vibrates between a first image and a second image later than the firstimage.

To suppress such vibration, the image processing device 1 according tothe first embodiment is used. Thus, the image is displayed on an imagedisplay device with the vibration suppressed.

Image Processing Device 1

Referring now to FIGS. 2 to 10, the configuration and process of theimage processing device 1 will be described. As shown in FIG. 2, theimage processing device 1 according to the first embodiment includes amotion estimator 2 that estimates the amount of motion of an objectbetween a first image and a second image later than the first image anda motion compensator 8 that performs a conversion process on the secondimage on the basis of the amount of motion of the object to suppress thevibration of the object between the first and second images. The motionestimator 2 has a first estimation mode in which the amount of motion ofthe object is estimated in a predetermined search area SA and a secondestimation mode in which the amount of motion of the object is estimatedon the basis of a larger area than the search area SA in the firstestimation mode. When a predetermined criterion is satisfied, the motioncompensator 8 performs the conversion process on the basis of the amountof motion estimated in the second estimation mode; when thepredetermined criterion is not satisfied, the motion compensator 8performs the conversion process on the basis of the amount of motionestimated in the first estimation mode.

The first embodiment will be described using consecutive two images, animage 1 and an image 2, as examples of the first image and the secondimage later than the first image.

The image processing device 1 includes the motion estimator 2, an areasetter 3, a tracking failure detector 5, a mode switch 7, and the motioncompensator 8. The motion estimator 2 includes a local area trackingunit 4 and a global motion estimator 6. The local area tracking unit 4includes a first motion estimator 41 and an image cutter 42. The globalmotion estimator 6 includes a second motion estimator 61 and a motioncorrector 62. These functions may be implemented by software orhardware. When implementing the functions by software, a CPU may becaused to execute a program. The program may be stored in built-inmemory or a computer-readable non-transitory storage medium. The programmay also be stored in external memory and then read so that thefunctions are implemented by so-called cloud computing. Whenimplementing the functions by hardware, ASIC, FPGA, DRP, or the like maybe used.

The area setter 3 sets an area corresponding to an object in a localmotion estimation mode, which is an example of a first estimation mode.In the first embodiment, the area corresponding to the object isreferred to as a local area LA (to be discussed later with reference toFIG. 4). In the local motion estimation mode, the local area LA istracked by estimating the amount of motion of the object between thefirst image and the later second image.

The motion estimator 2 estimates the amount of motion of the objectbetween the first image and the later second image. The first and secondimages may be consecutive images included in multiple images or may betwo non-consecutive images.

The local area tracking unit 4 tracks the local area LA in the localmotion estimation mode. The local motion estimation mode is a mode inwhich the amount of motion of the object is estimated in thepredetermined search area SA. This mode can save the calculationresources compared to when estimating the amount of motion of the objectusing the entire first and second images. Also, the local motionestimation mode is a mode in which the object can be tracked in thesearch area SA, and therefore allows for accurate estimation of themotion of the object compared to when estimating the amount of motion ofthe object using the entire first and second images.

The image cutter 42 cuts out parts of the first and second images andoutputs the cut-out image parts to the first motion estimator 41. Theimage cutter 42 determines the range of the part of the image 1 to becut out, on the basis of coordinates received from the area setter 3.

The first motion estimator 41 estimates the motion of the object in thesearch areas SA of the first and second images using the image partsreceived from the image cutter 42. In the first embodiment, the amountof motion of the object estimated by the first motion estimator 41 isreferred to as an amount of tracked motion. The first motion estimator41 then outputs the amount of tracked motion to the motion compensator8. The first motion estimator 41 also outputs, to the tracking failuredetector 5, an estimation error (to be discussed later with reference toFIG. 4) indicating a value calculated on the basis of the difference inluminance between the object included in the first image and apredetermined range included in the second image.

The global motion estimator 6 estimates the motion of the object in aglobal motion estimation mode, which is an example of a secondestimation mode. The global motion estimation mode is a mode in whichthe amount of motion of the object is estimated from the entire firstand second images. For this reason, the global motion estimation moderequires a larger amount of calculation than the local motion estimationmode, while it allows for estimation of the motion of the object thatcannot be coped with in the local motion estimation mode. In the globalmotion estimation mode, the term “object” refers to the entire image.

The second motion estimator 61 estimates the amount of motion of theobject using the entire first and later second images. In the firstembodiment, the amount of motion of the object estimated by the secondmotion estimator 61 is referred to as an amount of global motion. Thesecond motion estimator 61 then outputs the amount of global motion tothe motion corrector 62.

The motion corrector 62 calculates an amount of correction used whenperforming a conversion process on the second image, on the basis of theamount of global motion received from the second motion estimator 61.Such correction will be described later with reference to FIG. 10. Themotion corrector 62 then outputs the corrected amount of global motionto the motion compensator 8.

The tracking failure detector 5 detects that the first motion estimator41 has failed to track the local area LA.

The mode switch 7 switches between the local motion estimation mode andthe global motion estimation mode, which is an example of the secondestimation mode. When switching between the local motion estimation modeand global motion estimation mode, the mode switch 7 outputs a modeswitch signal to the local area tracking unit 4 and global motionestimator 6. Thus, the mode of the image processing device 1 isswitched.

The motion compensator 8 performs a conversion process on the secondimage on the basis of the amount of motion of the object to suppress thevibration of the object between the first and second images. When thepredetermined criterion is satisfied, the motion compensator 8 performsthe conversion process on the basis of the amount of motion estimated inthe second estimation mode; when the predetermined criterion is notsatisfied, it performs the conversion process on the basis of the amountof motion estimated in the first estimation mode. Thus, the input image2 is converted into an image 2′, which is then outputted as an outputimage. As a result, the vibration of the object between the first andsecond images is suppressed.

Flowchart

Next, referring to FIG. 3, the process flow of the image processingdevice 1 will be described.

S1: Set Local Area

First, in S1, the area setter 3 sets a local area LA. The imageprocessing device 1 is previously placed in the local motion estimationmode.

The setting of the local area LA by the area setter 3 will be describedwith reference to FIG. 4. In the first embodiment, as shown in FIG. 4A,an area surrounding an object (vehicle) in an image is defined as thelocal area LA. When setting the local area LA, the area setter 3receives coordinates (x,y) identifying the position of the object. Thearea setter 3 then sets the local area LA by generating coordinatesindicating an area having a predetermined size, around the receivedcoordinates. The area setter 3 may have any configuration. For example,it may be an input device, such as a keyboard, touchscreen, voice inputdevice, or gesture recognition device. Note that the area setter 3 neednot set the local area LA as described above. For example, the areasetter 3 may use, as the local area LA, an area obtained by previouslydividing an image into multiple areas and combining a predeterminednumber of areas around an area including the received coordinates. Forexample, the area setter 3 may use, as the local area LA, a total ofnine areas obtained by expanding an area including the receivedcoordinates vertically and horizontally by two areas. Also, thecoordinates of an object detected using any object detection technologymay be automatically inputted to the area setter 3.

The area setter 3 then outputs the coordinates indicating the local areaLA to the image cutter 42 shown in FIG. 2.

S2: Track Local Area

Then, in S2, the local area tracking unit 4 tracks the local area LA inthe search area SA.

A technique by which the local area tracking unit 4 tracks the localarea LA will be described with reference to FIGS. 4A to 4C. Assume thatthe input image 1 shown in FIG. 4A has been changed to the input image 2shown in FIG. 4C in the subsequent frame. In the input images 1, 2, thelocal area LA is positioned in the search area SA. The search area SA isa range in which the local area LA is tracked. In other words, in thelocal motion estimation mode, the local area LA is tracked in the searcharea SA. The position and size of the search area SA may be setarbitrarily. For example, a predetermined area around the local area LAmay be previously set as the search area SA.

The local area tracking unit 4 estimates the local area LA from theimage 2 by moving a fitting area FA in the search area SA as shown inFIG. 4B.

Specifically, the image cutter 42 shown in FIG. 2 cuts out the localarea LA from the input image 1 and the search area SA from the inputimage 2 on the basis of the received coordinates. The image cutter 42then outputs the cut-out images to the first motion estimator 41.

As shown in FIG. 4B, the first motion estimator 41 generates the fittingarea FA having the same size as the local area LA in the search area SA.The first motion estimator 41 then compares the difference in luminancebetween the fitting area FA and the local area LA included the inputimage 1. Specifically, the first motion estimator 41 obtains relativepositions in the fitting area FA, of pixels included in the fitting areaFA and relative positions in the local area LA, of pixels included inthe local area LA. The first motion estimator 41 then calculates thedifferences in luminance between pixels whose relative positionscorrespond to each other, with respect to all pixels included thefitting area FA and local area LA and sums up the differences. Forexample, this calculation is performed using “Σ|Y₀-Y₁|” where Σrepresents the sum of all pixels in the respective areas; Y0 representsthe luminance of a pixel included in the local area LA; and Y1represents the luminance of a pixel included in the fitting area FA. Inthe first embodiment, the sum of the differences corresponds to anestimation error. Instead of defining the sum of the differences itselfas an estimation error, a value obtained by multiplying the sum of thedifferences by a predetermined factor may be defined as an estimationerror. That is, in the first estimation mode, a value calculated on thebasis of the difference in luminance between the object (the local areaLA) included in the first image and the predetermined area (the fittingarea FA in the search area SA) included in the second image can bedefined as an estimation error.

The first motion estimator 41 sequentially performs this calculationwhile shifting the fitting area FA in the search area SA on apixel-by-pixel basis. Specifically, in the first embodiment, the firstmotion estimator 41 first sets the fitting area FA on the upper-leftedge of the search area SA and then calculates Σ|Y₀-Y₁| while shiftingthe fitting area FA on a pixel-by-pixel basis in the right direction ofthe figure. When the fitting area FA reaches the upper-right edge of thesearch area SA, the first motion estimator 41 returns the fitting areaFA to the initial position and then shifts it by one pixel in thedownward direction of the figure, that is, sets it in a lower positionthan the initial position by one line. The first motion estimator 41then repeatedly performs this calculation until the fitting area FAreaches the lower-right edge of the search area SA.

The first motion estimator 41 then makes a comparison among Σ|Y₀-Y₁| ofall positions and identifies the coordinates of a fitting area FA havingthe smallest value. The first motion estimator 41 then estimates an arearepresented by the coordinates, as an area corresponding to the localarea LA included in the input image 1. This is because the area havingthe smallest luminance difference is most likely to correspond to thelocal area LA included in the input image 1. Note that when the localarea LA does not change between the input images 1 and 2 at all, theestimation error is 0.

In this way, the local area tracking unit 4 tracks the movement of thelocal area LA.

S3: Detect Failure Flag

Then, in S3, the tracking failure detector 5 determines whether afailure flag has been detected. A failure flag is detected on the basisof the predetermined criterion.

For example, when the local area LA makes a large movement between thetwo images or when the luminance of the local area LA changessignificantly therebetween, the first motion estimator 41 may havedifficulty in tracking the local area LA. For this reason, in the firstembodiment, the predetermined criterion is previously set as a criterionindicating that tracking is difficult, and when this criterion issatisfied (that is, when the first motion estimator 41 has difficulty intracking the local area LA), the tracking failure detector 5 detects afailure flag and generates a failure flag signal indicating thattracking is difficult.

The predetermined criterion will be described below with reference toFIG. 5. FIG. 5A is a graph obtained by plotting the estimation error(Σ|Y₀-Y₁|) of the local area LA on the vertical axis and the number ofimage frames on the horizontal axis. Note that FIG. 5 shows a case inwhich the number of frames is 300, that is, a case in which images 1 to300 have been inputted as input images.

In the first embodiment, the predetermined criterion is that “theestimation error exceeds a predetermined value” or that “the estimationerror exceeds a predetermined multiple of the average value.” Morespecifically, the predetermined criterion is that “the estimation errorexceeds the predetermined value and exceeds the predetermined multipleof the average value.” As used herein, the term “predetermined value”refers to a threshold (absolute value) and tracking of the local area LAwill become difficult when the estimation error exceeds this value. Thepredetermined value may be any value. In the first embodiment, thethreshold is, for example, 25,000. The term “average value” refers tothe average of the amounts of motion of multiple images arranged inchronological order. In FIG. 5A, a thick line in the graph representsthe average amount of motion, and the value is about 5,000. The term“predetermined multiple of the average amount of motion” refers to acoefficient arbitrarily set by the user and is, for example, 3 in thefirst embodiment.

Specifically, in the first embodiment, when the estimation error exceeds25,000 and the estimation error exceeds 15,000, it is determined thattracking of the local area LA is difficult. In the example of FIG. 5,this criterion is satisfied when the frame number is 220. Also, as shownin FIG. 5B, when the frame number is 220, the tracking failure detector5 generates a failure flag signal. Then, as shown in FIG. 2, thetracking failure detector 5 outputs the failure flag signal to the modeswitch 7 and motion compensator 8.

When the determination result in S3 is “No”, that is, when no failureflag has been detected, the process proceeds to S4.

S4: Output Amount of Motion

Then, in S4, as shown in FIG. 2, the local area tracking unit 4 outputs,to the motion compensator 8, the amount of motion (the amount of trackedmotion) to perform motion compensation in the subsequent S5.

S5: Motion Compensation

Then, in S5, the motion compensator 8 performs a compensation process.

The processes in S4 and S5, that is, the local motion estimation modewill be described below with reference to FIGS. 6 and 7.

Local Motion Estimation Mode

FIG. 6 is a diagram showing processing of various types of data in thelocal motion estimation mode. In FIG. 6, image data inputted to oroutputted from the area setter 3, local area tracking unit 4, and motioncompensator 8 is arranged in chronological order.

First, the area setter 3 generates the coordinates of the local area LAby performing the process of S1 in FIG. 3.

Since the input image 1 is only one input image, tracking of the localarea LA is difficult. Accordingly, the image 1 is outputted as an outputimage as it is.

Then, when the image 2, which is the input image of the subsequentframe, is inputted, data indicating the image 2 is outputted to thelocal area tracking unit 4 along with data indicating the image 1. Then,the local area tracking unit 4 estimates the amount of motion betweenthe image 1 and image 2 (the amount of motion 1) and outputs the amountof motion 1 and the data indicating the image 2 to the motioncompensator 8. The motion compensator 8 generates an image 2′ byperforming a conversion process on the image 2 on the basis of theamount of motion 1 and outputs the image 2′ as an output signalsubsequent to the image 1.

The conversion process based on the amount of motion will be describedbelow with reference to FIG. 7. FIGS. 7A to 7F show states in which animage is displayed on an image display device 9, in which FIGS. 7A, 7C,and 7E show input images; and FIGS. 7B, 7D, and 7F show output images.In these figures, D1 to D3 represent the display ranges of images 1 to3, and D2′ and D3′ represent the display ranges of images 2′ and 3′.

In the first embodiment, the motion compensator 8 compensates for theposition of the image so that the local area LA is displayed in theapproximate center of the display surface on which the image isdisplayed. Specifically, the motion compensator 8 moves the image 1 onthe basis of the coordinates set by the area setter 3 and thecoordinates of the approximate center of the display surface. In thefirst embodiment, to simplify the description, a case in which thecoordinates of the approximate center of the display surface arepreviously inputted to the area setter 3 will be described. That is, inFIG. 7A, the local area LA is previously set in the approximate centerof the display surface.

As shown in FIGS. 7A and 7B, when the image 1 is inputted, the image 1is outputted as it is. An upper-left position P0 (x0,y0) of the localarea LA identifies the position of the local area LA included in theimage 1, and the compensation process is performed using the position P0as a reference.

Next, as shown in FIG. 7C, assume that the local area LA included in theimage 1 has been moved in the image 2 by (Δx1,Δy1). This amount ofmovement corresponds to the amount of tracked motion calculated when thelocal area tracking unit 4 tracks the local area LA. The local areatracking unit 4 stores the upper-left position P1 (x1,y1) of the movedlocal area LA. Also, as shown in FIG. 6, the local area tracking unit 4outputs the amount of motion 1 (Δx1,Δy1) to the motion compensator 8.

To suppress the vibration of the local area LA, the motion compensator 8moves the image 2 along with D2 by the amount of motion 1 (Δx1,Δy1). Inthe first embodiment, the right direction of FIG. 7 is defined as apositive x-axis direction, and the vertical direction is defined as apositive y-axis direction. This process is performed using P0 (x0,y0)and P1 (x1,y1). Specifically, this process is performed by rendering theentire image 2 such that the coordinates (x1,y1) of P1 match thecoordinates (x0,y0) of P0. In other words, the motion compensator 8suppresses the vibration of the local area LA by moving the image 2 bythe opposite phase of the amount of tracked motion. The above process isthe conversion process. Due to this conversion process, the image 2′ isgenerated and outputted as an output image. Since the image 2 is movedalong with D2, D2′, which is the range displayed on the image displaydevice 9, of the moved entire image, is smaller than D2. Note that ahatched portion in FIG. 7D represents an area in which no image isdisplayed. Instead of being hatched, this area may be displayed in blackor mosaic, or any image, any scaled-down image, or the like may bedisplayed therein.

If it is not desired to generate an area where no image is displayed asseen in FIG. 7D, such area (the hatched portion) can be eliminated byenlarging the range D2′ and moving the local area LA to the approximatecenter of the display surface.

Referring back to FIG. 6, the description of processing of various typesof data in the local motion estimation mode will be continued. When animage 3 is inputted, data indicating the image 3 is outputted to thelocal area tracking unit 4. While, for the convenience of description,FIG. 6 shows that the data indicating the image 2 and the amount ofmotion 1 are outputted to the local area tracking unit 4, these piecesof data are actually already held by the local area tracking unit 4.

The local area tracking unit 4 calculates the amount of motion 2 betweenthe image 2 and image 3. In the first embodiment, as shown in FIG. 7E,the amount of motion 2 is calculated as (Δx2,Δy2) from the upper-leftposition P2 (x2,y2) of the local area LA included in the image 3 and P1(x1,y1) in the image 2. In an example of FIG. 7E, Δy2 is a negativevalue. The coordinates (x2,y2) of P2 are uniquely identified using P0(x0,y0) and P1 (x1,y1). As shown in FIG. 6, the local area tracking unit4 outputs the amount of motion 2 (Δx2,Δy2) to the motion compensator 8.

As shown in FIG. 7F, the motion compensator 8 moves the image 3 alongwith D3 by the amount of motion 1 plus the amount of motion 2(Δx2+Δx1,Δy2+Δy1). This process is performed using P0 (x0,y0) and P2(x2,y2). Specifically, this process is performed by rendering the entireimage 3 such that the coordinates (x2,y2) of P2 match the coordinates(x0,y0) of P0. Thus, the motion compensator 8 generates an image 3′ andoutputs the image 3′ as an output image subsequent to the image 2′.

Then, when an image 4 is inputted, a process similar to those withrespect to the images 2 and 3 is repeated. This process is performedeach time one image is inputted until the input image of the last frameis inputted.

In the local motion estimation mode, the subsequent image is convertedin accordance with P0 (x0,y0). For this reason, instead of calculatingthe amount of motion 2 between the image 2 and image 3, the amount ofmotion between the images 1 and image 3 may be directly calculated bycomparing the image 1 and image 3. However, the local area LA is trackedwith higher accuracy by comparing the image 2 and image 3 than bycomparing the image 1 and image 3. The reason is that the luminance inthe local area LA included in actual images 1, 2, 3, and the likechanges from moment to moment and a smaller estimation error (Σ|Y₀-Y₁|)is obtained by comparing with an image as recent as possible.

As seen above, in the local motion estimation mode, the local area LAcan be tracked in the search area SA using pattern matching.

Referring back to FIG. 3, the description of the flow will be continued.

When the determination result in S3 is “Yes”, that is, a failure flag isdetected, the tracking failure detector 5 outputs a failure flag signalto the mode switch 7 and motion compensator 8, and the process proceedsto S6.

In S6, the mode switch 7 outputs a mode switch signal to the local areatracking unit 4 and global motion estimator 6 and switches the imageprocessing device 1 from the local motion estimation mode to the globalmotion estimation mode. Then, the amount of motion calculated in theglobal motion estimation mode is outputted to the motion compensator 8,and the process proceeds to S4 and S5 as in the local motion estimationmode.

Referring now to FIGS. 8 to 10, the processes of S4 to S6, that is, theglobal motion estimation mode will be described.

S6: Global Motion Estimation Mode

The global motion estimation mode is a mode in which the amount ofmotion of the object is estimated from the entire first and secondimages. In the global motion estimation mode, the coordinates of thelocal area LA are not set unlike in the local motion estimation mode.

As shown in FIG. 9, the global motion estimation mode does not have theconcept of the local area LA. For this reason, an entire image isdivided into multiple areas (9 areas in FIG. 9), and the direction inwhich the entire image is moving is estimated. That is, in the globalmotion estimation mode, the entire image is regarded as an object whosemotion is to be estimated, and the amount of motion of this object isestimated. In the first embodiment, description will be made using anexample in which an image is divided into nine divided areas.

The second motion estimator 61 estimates the movement direction of eachdivided area using pattern matching or a feature vector. As used herein,the term “movement direction of each divided area” refers to thedirection in which an object included in each divided area moves. Whileany estimation method may be used, a process similar to that withrespect to the local area LA, for example, may be performed on eachdivided area. Specifically, the movement direction of each divided areacan be estimated by setting the center of each divided area as a localarea LA, setting the outer circumference of the divided area as a searcharea SA, and performing a process similar to tracking of the local areaLA on each of the nine divided areas. The second motion estimator 61shown in FIG. 2 then estimates the most common direction of the movementdirections of the respective divided areas as the movement direction ofthe entire image. In an example of FIG. 9, the movement directions ofsix divided areas of the nine divided areas are an upper-right directionand therefore the entire image is estimated to be moving in theupper-right direction. The second motion estimator 61 then estimates theaverage of the amounts of motion of the respective divided areasconsidering the movement direction, as the amount of motion of theentire image. This amount of motion corresponds to the amount of globalmotion. Note that even if there is a divided area whose motion isdifficult to estimate, the amount of motion of the entire image can beestimated by calculating the average of the areas except for that area.

The significance of the motion corrector 62 will be described below withreference to FIG. 10. In the global motion estimation mode, thecoordinates of the local area LA are not set unlike in the local motionestimation mode. For this reason, the motion corrector 62 performs acorrection process so that the motion compensator 8 properly performs aconversion process.

FIGS. 10A, 10C, 10E, and 10G show input images, and FIGS. 10B, 10D, 10F,and 10H show output images. In FIGS. 10A to 10H, an area correspondingto the local area LA in FIG. 7 is referred to as an object OJT for theconvenience of description.

As shown in FIGS. 10A and 10B, also in the global motion estimationmode, a process similar to that in the local motion estimation mode isperformed on the image 1.

As shown in FIG. 10C, assume that the amount of motion 1 of the image 2with respect to the image 1 is (Δx1,Δy1). As shown in FIG. 8, the secondmotion estimator 61 outputs the amount of motion 1 (Δx1,Δy1) to themotion compensator 8 and motion corrector 62. As in the local motionestimation mode, the motion compensator 8 moves the image 2 along withD2 by the amount of motion 1 (Δx1,Δy1).

Next, as shown in FIG. 10E, assume that the amount of motion 2 of theimage 3 with respect to the image 2 is (Δx2,Δy2). OJT (10C) in FIG. 10Erepresents the position of the object OJT in FIG. 10C.

In the global motion estimation mode, the coordinates of the local areaLA are not set. For this reason, even if the motion compensator 8 movesthe image 3 by the amount of global motion between the image 2 and image3 in a conversion process, the image 3 is returned to the state of theimage 2 shown in FIG. 10C. For this reason, in the global motionestimation mode, the second most recent amount of global motion, thatis, the amount of global motion between the image 1 and image 2 is used.Thus, the display position of the object OJT in the image 3 can beapproximately matched with the display position of the object OJT in theimage 1. Specifically, the motion corrector 62 generates the correctedamount of motion 2′ (Δx1′(=Δx2+Δx1),Δy1′(=Δy2+Δy1)) by adding the amountof motion 1 (Δx1,Δy1) as a correction amount to the amount of motion 2(Δx2,Δy2). As shown in FIG. 8, the motion corrector 62 outputs thecorrected amount of motion 2′ (Δx1′,Δy1′) to the motion compensator 8.

The motion compensator 8 moves the image 3 along with D3 by thecorrected amount of motion 2′ (Δx1′,Δy1′). In FIG. 10E, Δy2 is anegative value. This process is performed by adding the corrected amountof motion 2′ to the coordinates of the entire image 3 and rendering theentire image 3. Thus, the motion compensator 8 generates an image 3′ andoutputs the image 3′ as an output image subsequent to the image 2′. Notethat (Δx2′,Δy2′) corresponds to the corrected amount of global motion inFIG. 2.

Next, as shown in FIG. 10G, assume that the amount of motion 3 of theimage 4 with respect to the image 3 is (Δx3,Δy3). OJT (10E) in FIG. 10Grepresents the position of the object OJT in FIG. 10E. By using thecorrected amount of motion 2′, which is the second most recent amount ofglobal motion, to perform a conversion process on the image 4, thedisplay position of the object OJT in the image 4 can be approximatelymatched with the display position of the object OJT in the image 1.Specifically, the motion corrector 62 generates the corrected amount ofmotion 3′ by adding the corrected amount of motion 2′ (Δx2′,Δy2′) as acorrection amount to the amount of motion 3 (Δx3,Δy3). Then, as shown inFIG. 8, the motion corrector 62 outputs the corrected amount of motion3′ (Δx3′(=Δx3+Δx2′),Δy3′(=Δy3+Δy2′)) to the motion compensator 8.

The motion compensator 8 moves the image 4 along with D4 by thecorrected amount of motion 3′ (Δx3′,Δy3′). This process is performed byadding the corrected amount of motion 3′ to the coordinates of theentire image 4 and rendering the entire image 4. Thus, the motioncompensator 8 generates an image 4′ and outputs the image 4′ as anoutput image subsequent to the image 3′. Note that (Δx3′,Δy3′)corresponds to the corrected amount of global motion in FIG. 2.

This process is performed each time one image is inputted until theinput image of the last frame is inputted.

As seen above, in the global motion estimation mode, the amount ofmotion of the object is estimated from the entire image using patternmatching.

Referring back to FIG. 3, the description of the flow will be continued.

S7: Determine Whether Current Image is Image of Last Frame

After the motion compensator 8 completes the compensation process, it isdetermined in S7 whether the current image is the image of the lastframe. When the determination result in S7 is “No”, that is, the currentimage is not the image of the last frame, the processes of S2 to S5 arerepeatedly performed. Even if the mode is switched to the global motionestimation mode in S6, the process returns to S2. As described above,this is because it is necessary to determine whether tracking of thelocal area LA has been allowed again. In this case, it can be said thatthe process of the local motion estimation mode is being performed inthe background, although the local motion estimation mode has beenswitched to the global motion estimation mode. For this reason, when thepredetermined criterion is not satisfied, the second estimation mode canbe switched to the first estimation mode.

On the other hand, when the determination result in S7 is “Yes”, thatis, the current image is the image of the last frame, the imageprocessing device 1 ends the vibration suppression process.

As described above, the image processing device 1 according to the firstembodiment determines in which of the local motion estimation mode andglobal motion estimation mode the motion compensator 8 performs theconversion process, on the basis of the predetermined criterion. Thus,when tracking of the local area LA is allowed, the image processingdevice 1 is able to perform image processing in the local motionestimation mode, in which calculation is small in amount. On the otherhand, when tracking of the local area LA is difficult, the local motionestimation mode is switched to the global motion estimation mode, inwhich the entire image is used. Thus, the image processing device 1 isable to efficiently suppress the vibration of the image while saving thecalculation resources.

Second Embodiment

Next, referring to FIGS. 11 and 12, an image processing device 1according to a second embodiment of the present invention will bedescribed. A main difference between the second embodiment and firstembodiment is that the global motion estimation mode is not used in thesecond embodiment.

The image processing device 1 according to the second embodimentincludes a first motion estimator 41 that estimates the amount of motionof a local area LA in a predetermined search area SA and a motioncompensator 8 that performs a conversion process on a second image sothat the vibration of the local area LA between a first image and thesecond image is suppressed, on the basis of the amount of motion of thelocal area LA. The first motion estimator 41 has a first estimation modein which the amount of motion of an image is estimated by tracking thelocal area LA in the search area SA and a second estimation mode inwhich the local area LA is tracked on the basis of a larger area thanthe search area SA in the first estimation mode.

As shown in FIGS. 11, S1 to S5 in the process flow of the secondembodiment are similar to those in the first embodiment and thereforewill not be described. S6 and S7 (the expansion of the search area SA)in FIG. 11 will be described below with reference to FIG. 12. Note thatin FIG. 12, a number added to the search area SA represents the numberof times of expansion of the search area SA. For example, a search areaSA0 shown in FIG. 12A and 12B represents a yet-to-be-expanded searcharea SA (corresponding to the search area SA in the first embodiment). Asearch area SA1 shown in FIG. 12C represents a search area SA that hasbeen expanded once.

A comparison is made between an input image shown in FIG. 12A and aninput image of the subsequent frame shown in FIG. 12B. In the inputimage shown in FIG. 12B, the local area LA is located outside the searcharea SA0. In this case, the minimum value of the estimation error(Σ|Y₀-Y₁|) in the search area SA0 is increased, and a predeterminedcriterion shown in FIG. 5 is no longer satisfied. In other words,tracking of the local area LA becomes difficult.

For this reason, the local area tracking unit 4 newly sets a search areaSA1 by expanding the search area SA0 using an area expander (not shown).This state is shown in FIG. 12C. That is, in the second embodiment, thelocal motion estimation mode after expanding the search area SAcorresponds to the second estimation mode.

Then, the local area LA is tracked by again performing the process shownin FIGS. 4A to 4C. Note that the size and position of the search areaSA1 is arbitrary and may be properly set so as to contain the SA0 inaccordance with the characteristics of the image.

In the second embodiment, the search area SA0 is expanded, and the firstestimation mode is switched to the second estimation mode. Thus, thelocal area LA which has been difficult to be tracked before beingexpanded can be tracked. If necessary, the search area SA may beexpanded predetermined times repeatedly. If the size of the expandedsearch area SA becomes equal to the size of the entire image, the localarea LA can be tracked accurately as long as the local area LA islocated in the image, although calculation is large in amount.

Next, referring to FIG. 11, a process flow according to the secondembodiment will be described. As shown in FIG. 11, S1 to S5 in theprocess flow of the second embodiment are similar to those in the firstembodiment and therefore will not be described.

Then, in S3, the tracking failure detector 5 determines whether afailure flag has been detected. When the determination result in S3 is“No”, that is, when no failure flag has been detected, the processproceeds to S5.

Then, in S5, the local area tracking unit 4 outputs the amount of motionto the motion compensator 8. Then, in S6, the motion compensator 8performs a conversion process. On the other hand, when the determinationresult in S3 is “Yes”, that is, a failure flag has been detected, theprocess proceeds to S6 to determine whether the number of times ofexpansion of the search area SA has reached the upper limit. Note thatthe upper limit number of times may be any number of times. Hereinafter,an example in which the upper limit number of times is three will bedescribed.

In the initial state, the number of times of expansion is zero and doesnot exceed the upper limit. Therefore, the determination result in S6becomes “No”. Then, the process proceeds to S7, and the local areatracking unit 4 sets a search area SA1 by expanding the search area SA0.Then, the process again proceeds to S3 through S2. When tracking of thelocal area LA is allowed due to the setting of the search area SA1, theprocess proceeds to S4. On the other hand, when tracking of the localarea LA is still difficult although the search area SA1 has been set,that is, when the predetermined criterion shown in FIG. 5 is satisfied,the process again proceeds to S6. This process is repeatedly performeduntil the number of times of expansion of the search area SA exceeds theupper limit.

When the number of times of expansion exceeds the upper limit in S6,specifically, when tracking of the local area LA is difficult although asearch area SA3 has been set, the determination result in S6 becomes“Yes”, and the process proceeds to S8. That is, when tracking of thelocal area LA is difficult although the search area SA has been expandeduntil the number of times of expansion exceeds the upper limit, theimage is displayed as it is without performing a compensation process.In the present embodiment, the upper limit number of times of expansionof the search area SA is set to three. However, if the size of thesearch area SA3 that has been expanded three times is set so as to beequal to the size of the entire image, the process proceeds to S4 afterthe upper limit number of times is exceeded in S6, and a compensationprocess can be performed.

Then, the process proceeds to S8, and a process similar to S7 in FIG. 3related to the first embodiment is performed.

When the determination result in S8 is “Yes”, that is, the current imageis the image of the last frame, the image processing device 1 ends thevibration suppression process.

As described above, the local area LA can be accurately tracked byexpanding the search area SA using the above configuration and process.Thus, it is possible to efficiently suppress the vibration of the imagewhile saving the calculation resources.

When the predetermined criterion is no longer satisfied in the secondestimation mode, for example, when the local area LA departs from thesearch area SA2 and is located in the immediately preceding search areaSA1, tracking of the local area LA may be allowed in the search areaSA1. In this case, the mode switch 7 may reduce the size of the searcharea SA by one level so that the local area LA is tracked. Also, whenthe local area LA is located in the initial search area SA, the secondestimation mode may be switched to the first estimation mode.

Third Embodiment

Next, referring to FIG. 13, an image processing device 1 according to athird embodiment of the present invention will be described. The imageprocessing device 1 according to the third embodiment performs a processobtained by combining the processes of the first and second embodiments.S1 to S7 in FIG. 13 are similar to those in the second embodiment andtherefore will not be described.

In the third embodiment, when the upper limit number of times isexceeded in S6, specifically, when tracking of the local area LA isdifficult although a search area SA3 has been set, the determinationresult in S6 becomes “Yes”, and the process proceeds to S8. In S8, themode is switched to the global motion estimation mode, as in the S6 ofFIG. 3 related to the first embodiment.

Then, the process proceeds to S5 and S9 through S4, and a processsimilar to S7 of FIG. 3 related to the first embodiment is performed.

On the other hand, when the determination result in S9 is “Yes”, thatis, the current image is the image of the last frame, the imageprocessing device 1 ends the vibration suppression process.

As described above, the image processing device 1 according to the thirdembodiment is able to accurately track the local area LA by expandingthe search area SA. Even when the number of times of expansion of thesearch area SA exceeds the upper limit, the motion compensator 8 cancontinue the conversion process by switching the mode to the globalmotion estimation mode.

Others

While the various embodiments have been described, the image processingdevice 1 according to the present invention is not limited thereto. Forexample, the criterion for switching between the first estimation modeand the second estimation mode may be a single criterion that “theestimation error exceeds the predetermined value.” Alternately, thecriterion may be a single criterion that “the estimation error exceedsthe predetermined multiple of the average value.” Also, the thresholdshown in FIG. 5 may be dynamically changed in accordance with avariation in the amount of motion of the image. For example, when theamount of motion of the image becomes equal to or greater than thepredetermined value multiple times continuously, the threshold may beincreased. In this case, the accuracy with which the local area LA istracked is improved even if the average value is not used as thepredetermined criterion.

The mode switch 7 may be omitted. In this case, the local motionestimation mode and global motion estimation mode may be performedsimultaneously, and the motion compensator 8 may make a selection onwhether it should perform the conversion process on the basis of theamount of motion estimated in the local motion estimation mode or on thebasis of the amount of motion estimated in the global motion estimationmode.

When estimating the amount of motion in the local motion estimation modeand global motion estimation mode, a feature vector may be used in placeof pattern matching.

The image processing apparatus 1 may be provided as a PC, a server orcomputer, or a set-top box connected to a display device. The imageprocessing device 1 may be incorporated into a display device. The imageprocessing device 1 may also be provided as a computer-readablenon-transitory storage medium having the functions of the imageprocessing device 1 implemented thereon. A program for implementing thefunctions of the image processing apparatus 1 may be distributed throughthe internet or the like.

The elements of the image processing device 1 may be contained in thesame case, or may be disposed in multiple cases in a distributed manner.

DESCRIPTION OF REFERENCE SIGNS

-   1: image processing device-   2: motion estimator-   3: area setter-   4: local area tracking unit-   41: first motion estimator-   42: image cutter-   5: tracking failure detector-   6: global motion estimator-   61: second motion estimator-   62: motion corrector-   7: mode switch-   8: motion compensator-   9: image display device-   LA: local area-   SA: search area-   FA: fitting area

The invention claimed is:
 1. An image processing device comprising: amotion estimating circuit configured to estimate the amount of motion ofan object between a first image and a second image later than the firstimage; a motion compensating circuit configured to perform a conversionprocess on the second image so that vibration of the object between thefirst image and the second image is suppressed, on the basis of theamount of motion of the object; and a mode switching circuit configuredto switch between a first estimation mode and a second estimation mode,wherein the motion estimating circuit has the first estimation mode inwhich the amount of motion of the object is estimated in a predeterminedsearch area and the second estimation mode in which the amount of motionof the object is estimated in a larger area than the predeterminedsearch area in the first estimation mode, when a predetermined criterionis satisfied, the motion compensating circuit performs the conversionprocess on the basis of the amount of motion estimated in the secondestimation mode, and the predetermined search area of the firstestimation mode is in the first image and is smaller than the firstimage, the larger area of the second estimation mode is in the firstimage and includes the predetermined search area of the first estimationmode, and when the predetermined criterion is not satisfied, the motioncompensating circuit performs the conversion process on the basis of theamount of motion estimated in the first estimation mode, in the firstestimation mode, the motion estimating circuit defines, as an estimationerror, a value calculated on the basis of a difference in luminancebetween the object included in the first image and a predetermined areaincluded in the second image, and the predetermined criterion is thatthe estimation error exceeds a predetermined value, or that theestimation error exceeds a predetermined multiple of an average value,or both.
 2. The image processing device of claim 1, wherein the secondestimation mode is a global motion estimation mode in which the amountof motion of the object is estimated from the entire first and secondimages.
 3. The image processing device of claim 1, wherein in the secondestimation mode, the amount of motion of the object is estimated in anew search area obtained by expanding a size of the search area in thefirst estimation mode.
 4. The image processing device of claim 1,wherein the motion estimator estimates the amount of motion of theobject using pattern matching or a feature vector.
 5. The imageprocessing device of claim 1, further comprising: an area settingcircuit configured to set an area corresponding to the object in thefirst estimation mode.
 6. The image processing device of claim 5,wherein the motion compensating circuit performs the conversion processon the second image so that the area set by the area setting circuit islocated in an approximate center of a display surface on which the firstand second images are displayed.
 7. The image processing device of claim1, wherein when the predetermined criterion is not satisfied, the modeswitching circuit switches the second estimation mode to the firstestimation mode.
 8. An image display device comprising the imageprocessing device of claim
 1. 9. An image processing program for causinga computer to function as: a motion estimating circuit configured toestimate the amount of motion of an object between a first image and asecond image later than the first image; a motion compensating circuitconfigured to perform a conversion process on the second image so thatvibration of the object between the first image and the second image issuppressed, on the basis of the amount of motion of the object; and amode switching circuit configured to switch between a first estimationmode and a second estimation mode, wherein the image processing programis stored in a computer-readable non-transitory storage medium, themotion estimating circuit has the first estimation mode in which theamount of motion of the object is estimated in a predetermined searcharea and the second estimation mode in which the amount of motion of theobject is estimated in a larger area than the predetermined search areain the first estimation mode, when a predetermined criterion issatisfied, the motion compensating circuit performs the conversionprocess on the basis of the amount of motion estimated in the secondestimation mode, and the predetermined search area of the firstestimation mode is in the first image and is smaller than the firstimage, the larger area of the second estimation mode is in the firstimage and includes the predetermined search area of the first estimationmode, and when the predetermined criterion is not satisfied, the motioncompensating circuit performs the conversion process on the basis of theamount of motion estimated in the first estimation mode, in the firstestimation mode, the motion estimating circuit defines, as an estimationerror, a value calculated on the basis of a difference in luminancebetween the object included in the first image and a predetermined areaincluded in the second image, and the predetermined criterion is thatthe estimation error exceeds a predetermined value, or that theestimation error exceeds a predetermined multiple of an average value,or both.
 10. An image processing device comprising: a motion estimatingcircuit configured to estimate the amount of motion of an object betweena first image and a second image later than the first image; and amotion compensating circuit configured to perform a conversion processon the second image so that vibration of the object between the firstimage and the second image is suppressed, on the basis of the amount ofmotion of the object, wherein the motion estimating circuit has a firstestimation mode in which the amount of motion of the object is estimatedin a predetermined search area and a second estimation mode in which theamount of motion of the object is estimated in a larger area than thepredetermined search area in the first estimation mode, thepredetermined search area of the first estimation mode is in the firstimage and is smaller than the first image, the second estimation mode isa global motion estimation mode, and the larger area of the globalmotion estimation mode match with the entire first image, in the globalmotion estimation mode, the larger area than the predetermined searcharea is divided into a plurality of divided areas, and movementdirection is estimated for each divided area, and the amount of motionof the object is estimated from the movement directions of the dividedareas, when a predetermined criterion is satisfied, the motioncompensating circuit performs the conversion process on the basis of theamount of motion estimated in the second estimation mode, and when thepredetermined criterion is not satisfied, the motion compensatingcircuit performs the conversion process on the basis of the amount ofmotion estimated in the first estimation mode.